R Notebook

Applied exercises

8. This exercise relates to the College data set, which can be found in the file College.csv. It contains a number of variables for 777 different universities and colleges in the US. The variables are:

• Private : Public/private indicator
• Apps : Number of applications received
• Accept : Number of applicants accepted
• Enroll : Number of new students enrolled
• Top10perc : New students from top 10 % of high school class
• Top25perc : New students from top 25 % of high school class
• F.Undergrad : Number of full-time undergraduates
• P.Undergrad : Number of part-time undergraduates
• Outstate : Out-of-state tuition
• Room.Board : Room and board costs
• Books : Estimated book costs
• Personal : Estimated personal spending
• PhD : Percent of faculty with Ph.D.’s
• Terminal : Percent of faculty with terminal degree
• S.F.Ratio : Student/faculty ratio
• perc.alumni : Percent of alumni who donate
• Expend : Instructional expenditure per student
• Grad.Rate : Graduation rate

Before reading the data into R, it can be viewed in Excel or a text editor.

1. Use the read.csv() function to read the data into R. Call the loaded data college. Make sure that you have the directory set to the correct location for the data.

library(ISLR)
head(College)

2. Look at the data using the fix() function. You should notice that the first column is just the name of each university. We don’t really want R to treat this as data. However, it may be handy to have these names for later. Try the following commands:

In the ISLR package the version of College has the row names already assigned as the unviersity names.

1. Use the summary() function to produce a numerical summary of the variables in the data set.
2. Use the pairs() function to produce a scatterplot matrix of the first ten columns or variables of the data. Recall that you can reference the first ten columns of a matrix A using A[,1:10].
3. Use the plot() function to produce side-by-side boxplots of Outstate versus Private.
4. Create a new qualitative variable, called Elite, by binning the Top10perc variable. We are going to divide universities into two groups based on whether or not the proportion of students coming from the top 10% of their high school classes exceeds 50 %.

library(ggplot2)
lapply(College, summary)

$Private
 No Yes 
212 565 

$Apps
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
     81     776    1558    3002    3624   48090 

$Accept
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
     72     604    1110    2019    2424   26330 

$Enroll
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
     35     242     434     780     902    6392 

$Top10perc
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   1.00   15.00   23.00   27.56   35.00   96.00 

$Top25perc
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    9.0    41.0    54.0    55.8    69.0   100.0 

$F.Undergrad
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    139     992    1707    3700    4005   31640 

$P.Undergrad
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    1.0    95.0   353.0   855.3   967.0 21840.0 

$Outstate
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   2340    7320    9990   10440   12920   21700 

$Room.Board
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   1780    3597    4200    4358    5050    8124 

$Books
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   96.0   470.0   500.0   549.4   600.0  2340.0 

$Personal
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
    250     850    1200    1341    1700    6800 

$PhD
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   8.00   62.00   75.00   72.66   85.00  103.00 

$Terminal
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   24.0    71.0    82.0    79.7    92.0   100.0 

$S.F.Ratio
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   2.50   11.50   13.60   14.09   16.50   39.80 

$perc.alumni
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   0.00   13.00   21.00   22.74   31.00   64.00 

$Expend
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   3186    6751    8377    9660   10830   56230 

$Grad.Rate
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  10.00   53.00   65.00   65.46   78.00  118.00 

$Elite
   Length     Class      Mode 
      777 character character 

pairs(College[1:10])

boxplot(College$Outstate ~ College$Private)

College$Elite <- ifelse(College$Top10perc > 50, "Yes", "No")

Use the summary() function to see how many elite universities there are. Now use the plot() function to produce side-by-side boxplots of Outstate versus Elite.

summary(College$Elite)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.0000  0.0000  0.0000  0.1004  0.0000  1.0000 

10% of colleges are ‘Elite’.

22. Use the hist() function to produce some histograms with differing numbers of bins for a few of the quantitative variables. You may find the command par(mfrow=c(2,2)) useful: it will divide the print window into four regions so that four plots can be made simultaneously. Modifying the arguments to this function will divide the screen in other ways.

hist(College$Enroll)

hist(College$F.Undergrad)

hist(College$PhD)

hist(College$Accept/College$Apps) # Per every application, barely anyone accepts a student.

hist(College$Grad.Rate[College$Private == "Yes"])

hist(College$Grad.Rate[College$Private == "No"])

6. Continue exploring the data, and provide a brief summary of what you discover.
   To do.

9. This exercise involves the Auto data set studied in the lab. Make sure that the missing values have been removed from the data.

1. Which of the predictors are quantitative, and which are qualitative?

lapply(Auto[-ncol(Auto)], summary)

$mpg
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   9.00   17.00   22.75   23.45   29.00   46.60 

$cylinders
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  3.000   4.000   4.000   5.472   8.000   8.000 

$displacement
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   68.0   105.0   151.0   194.4   275.8   455.0 

$horsepower
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   46.0    75.0    93.5   104.5   126.0   230.0 

$weight
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   1613    2225    2804    2978    3615    5140 

$acceleration
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   8.00   13.78   15.50   15.54   17.02   24.80 

$year
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  70.00   73.00   76.00   75.98   79.00   82.00 

$origin
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  1.000   1.000   1.000   1.577   2.000   3.000 

2. What is the range of each quantitative predictor? You can answer this using the range() function.

lapply(Auto[c("mpg", "displacement", "horsepower", "weight", "acceleration", "year")], range)

$mpg
[1]  9.0 46.6

$displacement
[1]  68 455

$horsepower
[1]  46 230

$weight
[1] 1613 5140

$acceleration
[1]  8.0 24.8

$year
[1] 70 82

3. What is the mean and standard deviation of each quantitative predictor?

lapply(Auto[c("mpg", "displacement", "horsepower", "weight", "acceleration", "year")], mean)

$mpg
[1] 23.44592

$displacement
[1] 194.412

$horsepower
[1] 104.4694

$weight
[1] 2977.584

$acceleration
[1] 15.54133

$year
[1] 75.97959

4. Now remove the 10th through 85th observations. What is the range, mean, and standard deviation of each predictor in the subset of the data that remains?

basic <- function(x) {
  summary <- c(mean = mean(x, na.rm = T), range = range(x, na.rm = T), sd = sd(x, na.rm = T))
  summary
}
lapply(Auto[-(10:85), -ncol(Auto)], basic)

$mpg
     mean    range1    range2        sd 
24.404430 11.000000 46.600000  7.867283 

$cylinders
    mean   range1   range2       sd 
5.373418 3.000000 8.000000 1.654179 

$displacement
     mean    range1    range2        sd 
187.24051  68.00000 455.00000  99.67837 

$horsepower
     mean    range1    range2        sd 
100.72152  46.00000 230.00000  35.70885 

$weight
     mean    range1    range2        sd 
2935.9715 1649.0000 4997.0000  811.3002 

$acceleration
     mean    range1    range2        sd 
15.726899  8.500000 24.800000  2.693721 

$year
     mean    range1    range2        sd 
77.145570 70.000000 82.000000  3.106217 

$origin
    mean   range1   range2       sd 
1.601266 1.000000 3.000000 0.819910 

5. Using the full data set, investigate the predictors graphically, using scatterplots or other tools of your choice. Create some plots highlighting the relationships among the predictors. Comment on your findings.

plot(Auto[1:ncol(Auto) - 1])

It’s interesting to find that mpg is negatively related to displacement, horsepower and weight in a similar way. In fact, the correlation are:

correlations <- c(first = cor(Auto$mpg, Auto$displacement), second = cor(Auto$mpg, Auto$horsepower), third = cor(Auto$mpg, Auto$weight))
correlations

     first     second      third 
-0.8051269 -0.7784268 -0.8322442 

All within a 5 figure difference. and above -0.70. In contrast, the correlations between displacement, horsepower and weight are strong and positive.

correlations2 <- c(displa_horse = cor(Auto$displacement, Auto$horsepower), displace_weight = cor(Auto$displacement, Auto$weight), horsep_weight = cor(Auto$horsepower, Auto$weight))
correlations2

   displa_horse displace_weight   horsep_weight 
      0.8972570       0.9329944       0.8645377 

plot(Auto$displacement, Auto$horsepower)

plot(Auto$displacement, Auto$weight)

plot(Auto$horsepower, Auto$weight)

6. Suppose that we wish to predict gas mileage (mpg) on the basis of the other variables. Do your plots suggest that any of the other variables might be useful in predicting mpg? Justify your answer.

Yeah, just as we saw in the matrix scatterplot, mpg is highly correlated with several variables. Not only there’s a visual pattern, but the correlation coefficients are substantial.

1. To begin, load in the Boston data set.

library(MASS)
head(Boston)

How many rows are in this data set? How many columns? What do the rows and columns represent?

dim(Boston)

[1] 506  14

The data has 506 rows and 14 columns. Each row represents housing values in suburbs from Boston. See help(Boston).

2. Make some pairwise scatterplots of the predictors (columns) in this data set. Describe your findings.

plot(Boston)

Well, I find interesting that there’s some clear non-linear patterns such as in nox ~ dis as well as Istat ~ medv and rm ~ Istat.

3. Are any of the predictors associated with per capita crime rate? If so, explain the relationship.

crim is correlated with age, div, lstat and medv. The relationships are not straightforward though.

for (i in c("age", "dis", "lstat", "medv")) {
plot(Boston[, i], Boston[,"crim"], xlab = paste(i))
}

As the proportion of owner-occupied units built prior to 1940 increases, crime rates increase. But note that there are threshold effects. At around 40% of age, crime rates start to increase and only at around 80% it becomes substantial. This relationship might not be that strong linearly, but once you account for non-linearities it might be significant.

With dis (distance to five Boston employment centres), crime rates are higher only when employment centers are close, but as it passes increases over 4, crime rates set at 0.

With lstat it’s similar as with age, crime rates increases only after some value, more or less around 15. It seems that crime rates are not linearly related to almost any of these variables. The same pattern can be found with medv.

4. Do any of the suburbs of Boston appear to have particularly high crime rates? Tax rates? Pupil-teacher ratios? Comment on the range of each predictor.

library(car)
Boxplot(Boston$crim)

 [1] 381 419 406 411 415 405 399 428 414 418

Above the graph we can see the outliers which have significantly higher crime rates than the overall distribution. 10 suburbs, so not a small bunch.

Boxplot(Boston$ptratio)

 [1] 197 198 199 258 259 260 261 262 263 264

Boxplot(Boston$tax)

ptratio does have some outliers, but they’re not the same ones in crime rates. Surprisingly, tax doesn`t have any outliers, and the distribution is pretty smooth.

for (i in c("crim", "ptratio", "tax")) {
  print(range(Boston[, i]))
}

[1]  0.00632 88.97620
[1] 12.6 22.0
[1] 187 711

The range of crime rates can be as high as 88% and as low as 0%, quite some diversity between suburbs. Student-teacher ratio is much smaller, with a maximum of 22 students and a minimum of 13. Tax ranges from 187 to 711.

5. How many of the suburbs in this data set bound the Charles river?

table(Boston$chas)

  0   1 
471  35 

35 suburs.

6. What is the median pupil-teacher ratio among the towns in this data set?

median(Boston$ptratio)

[1] 19.05

19 pupils per teacher.

7. Which suburb of Boston has lowest median value of owner- occupied homes? What are the values of the other predictors for that suburb, and how do those values compare to the overall ranges for those predictors? Comment on your findings.

Boston[Boston$medv == min(Boston$medv), ]

Actually, both suburb 399 and 406 have the same lowest median value, which is 5.

First, both suburbs have virtually the same values for all predictors except for the crim, lstat and black variables. The crim variable is quite different with the second suburb having a really high crime rate compared to the maximum crime rate of 88% which is only happening in one suburb. The proportion of residential land zoned for lots over 25k sq.ft is 0 for both suburs, and the maximum is 100%. So no land zoned for lots in those suburbs. The indus variable is reasonably high with 18% while the minimum is 0% and the maximum is 27%. Both suburbs are not close to the Charles River and have high nitrogen oxide concentration at about 69% with the maximum being 87% and the minimum 38%.

Their number of rooms per dwelling is pretty average at 5.4/5.6 while the min is 3 and the max 9. Both suburbs are completely occupied by units built prior to 1940 and are reasonably close to five Boston employment centres (1.4 when the minimum is 1 and the highest 12.) Also both are at the maximum index of accesibility to radial highways, and that’s reasonably high as the minimum is 1 and they’re at 24.

The pupil-teacher ratio is 20.2 and the minimum is 12. As expected, both suburbs are highly populated by blacks, with one of the suburbs actually being the maximum number in the distribution. The lower status of the population is not unusually high, with the maximum and minimum being 37.9 and 1.73 and both suburbs are 30.5 and 22.9.

8. In this data set, how many of the suburbs average more than seven rooms per dwelling? More than eight rooms per dwelling? Comment on the suburbs that average more than eight rooms per dwelling.

table(Boston$rm > 7)

FALSE  TRUE 
  442    64 

table(Boston$rm > 8)

FALSE  TRUE 
  493    13 

Only 64 suburbs average above seven rooms, and only 13 average above eight rooms.

There are some outliers in these suburbs but they all have high age values, so a lot of owner occupied units prior to 1940. Also, all have high black density within their towns. All are far from the Charles River except suburb 164, and the proportion of residential land zone for lots over 25,000 sq.ft is zero with the exception of suburb 205.